Using Weighted Oriented Optical Flow Histograms for Multimodal Speaker Diarization

Speaker diarization currently focuses on using audio features to partition an audio stream into speaker homogeneous speech regions, in other words to determine “who spoke when”. Recent speaker diarization corpora contains video recordings in addition to the commonly used audio. Thus, we investigated the benefits of incorporating video features, namely histograms of weighted oriented optical flow, into a state-of-the-art diarization system. We found that although the video features did not perform well alone, after combining systems using audio and video features, we were able to improve the diarization error rate by 14% as compared to a speaker diarization system trained on audio-only features.